Add ORDS v0.3 repair data export endpoint - #58
Conversation
Groups store country_code as alpha-2 but the Open Repair Data Standard requires alpha-3, and Fixometer::getAllCountries only maps alpha-2 to a localised display name. Checked in as a static map rather than pulling in league/iso3166 so the exporter stays dependency-free for other instances. The key set is verified to match lang/instances/base/en/countries.php.
devices.problem is unsanitised free text with no Purify mutator, so it can carry raw HTML, email addresses, phone numbers and URLs with tracking parameters. Redaction is a standalone pass rather than logic inside the mapper so it can be toggled, tested against fixtures and audited on its own, and it reports per-run counts by type so an export can be checked before handover. Every pass fails closed: a regex error yields an empty string rather than leaking the unredacted original. The phone heuristic uses a nine-digit floor to keep hyphenated substance such as rpm ranges, part numbers and firmware versions intact.
Emits the fourteen standard columns in spec order. Most map straight across because the standard was derived from this schema, but three need work: year_of_manufacture is not stored so it is derived from the event year minus the item age, country needs alpha-3, and product_category_id needs a name lookup because our idcategories do not match the published ones. Vocabulary handling follows the Open Repair Alliance's published releases rather than the table schema, which is stale in two places: the standard collapses our screen-size and laptop-size splits into single categories, repair_status carries Unknown as a real value, and the barrier wording drops the "the" we seed.
Holds the instance identity and the vocabulary maps in config rather than a database table so a change shows up in a diff and is versioned alongside the standard it tracks. id_prefix and data_provider are deliberately undefaulted. The identifier is a stable key the consumer upserts on across releases, so serving under an unassigned or borrowed namespace would overwrite another provider's records; the endpoint refuses to serve while either is blank.
Adds GET /api/public/v2/repairs behind a new repairs:read scope, serving JSON or CSV with filters for update time, event window, powered category and pagination. The per-page ceiling is 1000 rather than the events endpoint's 100 because this is a bulk export rather than a display API. Visibility mirrors PublicEventController: approved events on approved groups, soft-deleted rows excluded, and allowed_network_ids honoured. The network restriction is a subquery rather than a join so a group belonging to several permitted networks does not multiply its devices. The feature flag moves from the shared public/v2 group onto each scope so the export can ship dark without taking the live events API down with it. CSV output escapes cells opening with a formula character, which spreadsheet software would otherwise execute on open, and carries the pagination and sync metadata in headers since the format has no envelope for it. Documented in docs/public-repairs-api.md alongside the events API.
repairs:read is the first scope beyond events:read, so the option help now lists the valid values instead of only showing the default.
Covers the auth surface including an events:read token being refused on repairs and the reverse, the feature flag in both directions, the config guards, all fourteen columns against a fixture, CSV column order and formula escaping, visibility exclusions, network restriction, filters, pagination and the redaction passes.
Ships every value blank or off. The identity settings are per deployment rather than defaulted in the chart, and the endpoint refuses to serve until an instance sets them.
8179e34 to
3618e63
Compare
|
QA 👍
|
|
1. The new OPTIONS guard never runs, and preflight stopped 404ing
Before this PR the group ran Nothing covers it. config([
'restarters.features.public_events_api' => false,
'restarters.features.public_repairs_api' => false,
]);
$this->options('/api/public/v2/repairs')->assertStatus(404);Route-level middleware won't fix it, since group middleware merges in front. The check has to go in the group list ahead of ->middleware(['publicApiEnabled', 'publicApiCors'])with 2.
$raw = $request->input('powered');
if ($raw === null || $raw === '') {
return;
}
$normalised = filter_var($raw, FILTER_VALIDATE_BOOL, FILTER_NULL_ON_FAILURE);Four non-blocking notes: CORS headers, the chart's problem-text default, the error body, the categories joinA browser can't read the CSV pagination headers
A question: the chart ships
|
PublicApiCors answers every OPTIONS request before calling $next, and group middleware merges ahead of the route action, so the flag check on the OPTIONS closure never ran. On an instance with both scopes off, preflight answered 204 with Access-Control-Allow-* headers where it previously returned 404. Moving the check into a group middleware ahead of PublicApiCors restores that, and leaves the closure as a bare noContent().
filter_var maps both "" and null to false rather than firing FILTER_NULL_ON_FAILURE, so a request for ?powered= merged false, passed the nullable|boolean rule, and narrowed the export to unpowered items. The docs tell callers to omit the parameter for both datasets, and an empty parameter is how plenty of clients spell an omitted one.
The CSV response carries its pagination metadata in X-Total-Count, X-Page, X-Per-Page, X-Last-Page and X-Max-Updated-At, and the docs advertise both those headers and CORS. Without an expose list a cross-origin caller receives the body and none of the headers, so it cannot tell when it has reached the last page. This also applies to the events endpoint, which shares the middleware.
Enabling the export and publishing volunteer free text were a single switch, so flipping FEATURE__PUBLIC_REPAIRS_API also published the problem column. The scrubber removes contact details and identifiers but cannot remove personal names, and the licensing question on that text is still open, so it now takes a deliberate opt-in. The config default moves with the chart, otherwise an instance that never sets the variable still publishes the column.
|
One line got left behind: docs/public-repairs-api.md#L28 still says |
The chart and config now default ORDS_INCLUDE_PROBLEM to false, but the docs still described the old default and told readers to set false for a structured-fields-only export, which is now what they get by default. A partner following them would expect the problem column and not receive it. The column table also listed problem unconditionally, so it now says which flag emits it and that thirteen columns is the default shape.
|
Send it 🚀 |
Description
Adds
GET /api/public/v2/repairs, which exports repair records in Open Repair Data Standard v0.3 for the Open Repair Alliance. They asked for this directly since they currently hand-export from the database. No exporter exists in this fork or upstream, so this is the first codified one. Ships behindFEATURE__PUBLIC_REPAIRS_API, off by default.Changes
GET /api/public/v2/repairsunder a newrepairs:readscope, JSON or CSV, withupdated_since,event_start,event_end,powered,pageandper_pagefilters. Per-page ceiling is 1000 rather than the events endpoint's 100 since this is a bulk export. Visibility mirrorsPublicEventController: approved events on approved groups, soft-deletes excluded,allowed_network_idshonoured. The network filter is a subquery rather than a join, so a group in several permitted networks doesn't multiply its devices. Files:API/PublicRepairController.php,EnsurePublicRepairsApiEnabled.php.public/v2group onto each scope, so repairs can ship dark without taking the live events API down with it. Files:routes/api.php,bootstrap/app.php,config/restarters.php.year_of_manufactureisn't stored so it's derived from the event year minusdevices.age,countryconverts alpha-2 to alpha-3, andproduct_category_idis a name lookup because ouridcategoriesdon't match theirs. Files:Services/Ords/OrdsRecordMapper.php,Helpers/Iso3166.php.config/ords.phpfollows ORA's published data rather than theirtableschema.json, which is stale. The standard collapses our screen-size and laptop-size splits, carriesUnknownas a realrepair_status, and drops the "the" from the barrier we seed.devices.problemis free text with no Purify mutator.ProblemTextScrubberstrips HTML and redacts emails, phone numbers, digit runs of 8 or more, and URL query strings, logging counts by type per request. Names aren't pattern-detectable and aren't removed.ORDS_ID_PREFIXandORDS_DATA_PROVIDERhave no defaults and the endpoint 503s while either is blank. The id is a stable key ORA upserts on across releases, so a borrowed namespace would overwrite another provider's rows.docs/public-repairs-api.md, alongside the existing events one.Output was validated against ORA's published
tableschema.json. Column names and order match exactly and every declared constraint passes, except theiridregex, which uses a hyphen while all 305,649 of their own published rows use an underscore.Deferred:
updated_sincedoesn't catch approval flips, so approving an old event leaves it invisible to an incremental consumer. Thecategoriesinner join silently drops devices with an orphanedcategory. Pagination is offset-based so a full crawl is quadratic. Throttling runs after auth, so invalid tokens hit the database unrated.QA Notes
We'll need to enable the flag on the test cluster and validate a full export against the spec.